{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "p5b0PzpW1xJq"
      },
      "source": [
        "![nebullvm nebuly AI accelerate inference optimize DeepLearning](https://user-images.githubusercontent.com/38586138/201391643-a80407e5-2c28-409c-90c9-327795cd27e8.png)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Accelerate PyTorch ResNet50 with Speedster"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "T9xuwZEHzN2K"
      },
      "source": [
        "Hi and welcome 👋\n",
        "\n",
        "In this notebook we will discover how in just a few steps you can speed up the response time of deep learning model inference using Speedster app from the open-source library `nebullvm`.\n",
        "\n",
        "We will\n",
        "1. Install Speedster and the deep learning compilers used by the library.\n",
        "2. Speed up a PyTorch ResNet50 without any loss of accuracy.\n",
        "3. Achieve faster acceleration on the same model by applying more aggressive optimization techniques (e.g. pruning, quantization) under the constraint of sacrificing up to 2% accuracy.\n",
        "\n",
        "Let's jump to the code."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_0ZRCXCR9693",
        "outputId": "19096862-5c5c-4f9f-b2ad-3ce084ccf213"
      },
      "outputs": [],
      "source": [
        "%env CUDA_VISIBLE_DEVICES=0"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "HbFy2Aykz2Qo"
      },
      "source": [
        "### Installation"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "ZPJHVZ74d8r2"
      },
      "outputs": [],
      "source": [
        "!pip install speedster"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "b0CLgQqxyrQi"
      },
      "source": [
        "Let's now import install the deep learning compilers used by Speedster that are not yet installed on the hardware.\n",
        "\n",
        "The installation of the compilers may take a few minutes."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GvK9mZSjeLU5"
      },
      "outputs": [],
      "source": [
        "!python -m nebullvm.installers.auto_installer --frameworks torch --compilers all"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "N5RXHoZl0p3p"
      },
      "source": [
        "## Optimization example with Pytorch"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-Ju-VcRH01Mw"
      },
      "source": [
        "In the following example we will try to optimize a standard resnet50 loaded directly from torchvision.\n",
        "\n",
        "Speedster can accelerate neural networks without loss of a user-defined precision metric, e.g. accuracy, or can achieve faster acceleration by applying more aggressive optimization techniques, such as pruning and quantization, that may have a negative impact on the selectic metric. The maximum threshold value for accuracy loss is determined by the metric_drop_ths parameter. Read more in the [docs](https://docs.nebuly.com/modules/speedster/getting-started).\n",
        "\n",
        "Let first test the optimization without accuracy loss (metric_drop_ths=0, default value), and then apply further accelerate it under the constrained of losing up to 2% of accuracy (metric = \"accuracy\", metric_drop_ths = 0.02)."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "skxEuemn171G"
      },
      "source": [
        "### Scenario 1 - No accuracy drop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "wVRLXrDi2VaG"
      },
      "source": [
        "First we load the model and optimize it using the Speedster API:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2RbgGruAeQcf"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision.models as models\n",
        "from speedster import optimize_model, save_model, load_model\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "# Load a resnet as example\n",
        "model = models.resnet50().to(device)\n",
        "\n",
        "# Provide an input data for the model    \n",
        "input_data = [((torch.randn(1, 3, 256, 256), ), torch.tensor([0]))]\n",
        "\n",
        "# Run Speedster optimization\n",
        "optimized_model = optimize_model(\n",
        "  model, input_data=input_data, optimization_time=\"unconstrained\"\n",
        ")\n",
        "\n",
        "# Try the optimized model\n",
        "x = torch.randn(1, 3, 256, 256).to(device)\n",
        "model.eval()\n",
        "res_optimized = optimized_model(x)\n",
        "res_original = model(x)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JMiuufyu2gD3"
      },
      "source": [
        "We can print the type of the optimized model to see which compiler was faster:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "ifuLyQsM9697",
        "outputId": "c1534e0d-e5bb-4d44-91e9-652593751d52"
      },
      "outputs": [],
      "source": [
        "optimized_model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4WxcxrUC9698"
      },
      "source": [
        "In our case, the optimized model type was PytorchTensorRTInferenceLearner, so this means that Pytorch-TensorRT was the faster compiler."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "iwHKfT349698"
      },
      "source": [
        "After the optimization step, we can compare the optimized model with the baseline one in order to verify that the output is the same and to measure the speed improvement"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "-IMJpfcb9698"
      },
      "source": [
        "First of all, let's print the results"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "uI8Kd1Z49698",
        "outputId": "832d3053-d6c8-4cc2-9b48-a59dfaa45d33"
      },
      "outputs": [],
      "source": [
        "res_original"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0I_zSpv29698",
        "outputId": "a0ba566d-6730-4954-8dd0-eb47b549cbf1"
      },
      "outputs": [],
      "source": [
        "res_optimized"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hBEtrYOd9699"
      },
      "source": [
        "Then, let's compare the performances:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "GqxiCAbpfcwV"
      },
      "outputs": [],
      "source": [
        "from nebullvm.tools.benchmark import benchmark"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "_0b0Bzwq-czD"
      },
      "outputs": [],
      "source": [
        "# Set the model to eval mode and move it to the available device\n",
        "\n",
        "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
        "\n",
        "model.eval()\n",
        "model.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "UqxzStjD2v0r"
      },
      "source": [
        "Here we compute the average throughput for the baseline model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "dkt67_Orwlv4",
        "outputId": "fc10c03c-c3ad-44d4-9fd6-c9b6dc0256c7"
      },
      "outputs": [],
      "source": [
        "benchmark(model, input_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "AgOv-GqQ3KIC"
      },
      "source": [
        "Here we compute the average throughput for the optimized model:\n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "4PodpaDVfwzT",
        "outputId": "27a42560-93a2-4c19-e68d-360093fe914c"
      },
      "outputs": [],
      "source": [
        "benchmark(optimized_model, input_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tBeRKNTI3iyK"
      },
      "source": [
        "## Scenario 2 - Accuracy drop"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "w3wutIzfAMe_"
      },
      "source": [
        "In this scenario, we set a max threshold for the accuracy drop to 2%"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "fO1nGqpj3p7z"
      },
      "outputs": [],
      "source": [
        "import torch\n",
        "import torchvision.models as models\n",
        "from speedster import optimize_model\n",
        "\n",
        "# Load a resnet as example\n",
        "model = models.resnet50().to(device)\n",
        "\n",
        "# Provide 100 random input data for the model  \n",
        "input_data = [((torch.randn(1, 3, 256, 256), ), torch.tensor([0])) for _ in range(100)]\n",
        "\n",
        "# Run Speedster optimization\n",
        "optimized_model = optimize_model(\n",
        "  model, input_data=input_data, optimization_time=\"unconstrained\", metric=\"accuracy\", metric_drop_ths=0.02\n",
        ")\n",
        "\n",
        "# Try the optimized model\n",
        "x = torch.randn(1, 3, 256, 256).to(device)\n",
        "res = optimized_model(x)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "qFKHaHM6-GKm"
      },
      "outputs": [],
      "source": [
        "# Set the model to eval mode and move it to the available device\n",
        "\n",
        "model.eval()\n",
        "model.to(device)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yfW9kmHX-pGi"
      },
      "source": [
        "Here we compute the average throughput for the baseline model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "0MMrL3959hli",
        "outputId": "2e8d27ec-a9f3-4f70-8c75-a0df974f2653"
      },
      "outputs": [],
      "source": [
        "benchmark(model, input_data)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "i3GqasOM-u8f"
      },
      "source": [
        "Here we compute the average throughput for the optimized model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "_IbAW0KA4Fm5",
        "outputId": "48d83c89-5687-42aa-a3b8-6989bcb66aa6"
      },
      "outputs": [],
      "source": [
        "benchmark(optimized_model, input_data)"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "ceb60d8c",
      "metadata": {
        "id": "ceb60d8c"
      },
      "source": [
        "## Save and reload the optimized model"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "d9eda1a0",
      "metadata": {},
      "source": [
        "We can easily save to disk the optimized model with the following line:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "62b6fcbf",
      "metadata": {},
      "outputs": [],
      "source": [
        "save_model(optimized_model, \"model_save_path\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "3c968d51",
      "metadata": {},
      "source": [
        "We can then load again the model:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "id": "c1340c49",
      "metadata": {},
      "outputs": [],
      "source": [
        "optimized_model = load_model(\"model_save_path\")"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "id": "b77ff2ac",
      "metadata": {
        "id": "b77ff2ac"
      },
      "source": [
        "<center> \n",
        "    <a href=\"https://discord.com/invite/RbeQMu886J\" target=\"_blank\" style=\"text-decoration: none;\"> Join the community </a> |\n",
        "    <a href=\"https://nebuly.gitbook.io/nebuly/welcome/questions-and-contributions\" target=\"_blank\" style=\"text-decoration: none;\"> Contribute to the library </a>\n",
        "</center>\n",
        "\n",
        "<center> \n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#key-concepts\" target=\"_blank\" style=\"text-decoration: none;\"> How speedster works </a> •\n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#documentation\" target=\"_blank\" style=\"text-decoration: none;\"> Documentation </a> •\n",
        "    <a href=\"https://github.com/nebuly-ai/nebullvm/tree/main/apps/accelerate/speedster#quick-start\" target=\"_blank\" style=\"text-decoration: none;\"> Quick start </a> \n",
        "</center>"
      ]
    }
  ],
  "metadata": {
    "accelerator": "GPU",
    "colab": {
      "collapsed_sections": [],
      "provenance": []
    },
    "gpuClass": "standard",
    "kernelspec": {
      "display_name": "Python 3.8.10 64-bit",
      "language": "python",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.8.9 (default, Apr 13 2022, 08:48:06) \n[Clang 13.1.6 (clang-1316.0.21.2.5)]"
    },
    "vscode": {
      "interpreter": {
        "hash": "31f2aee4e71d21fbe5cf8b01ff0e069b9275f58929596ceb00d14d90e3e16cd6"
      }
    }
  },
  "nbformat": 4,
  "nbformat_minor": 5
}
